Artificial Neural Network based Curve

We use artificial neural networks to perform curve prediction. For that, we have created a class of neural networks (feed forward multilayer perceptron networks with backpropagation) that have a topology which is determined by their genetic makeup. Using a simple evolutionary strategy on their genes, we optimise the networks’ topologies to solve the problems at hand. Using this approach, we could generate networks that are able to predict simple functions such as sin(x) or linear combinations thereof, with moderate computational overhead. However, it was not possible to generate networks that predict more complex functions such as sinc(x) or the NASDAQ composite index satisfactorily, within the allowed sizes for the networks. In general though, it appears to be a useful approach to generate neural networks using this form of evolutionary strategy as it substitutes for experience in neural network design. Introduction Curve prediction is one of the most popular applications for artificial neural networks. However, the success of using a neural network to solve a certain problem is inherently linked to the designer’s ability to apply an appropriate network to the task. Even relatively simple artificial neural networks such as the multi-layer-perceptron or variants thereof have several degrees of freedom, e.g. the number of neurons, the number of hidden layers, the type of transfer functions employed, which the network is very sensitive to. For most tasks there is no methodology for designing a neural network which guarantees success. Instead, we try to evolve a neural network topology that is suitable for any curve prediction task. Aim • To develop an evolvable artificial neural network representation • To optimise such a neural network to solve a number of curve prediction • To evaluate the ability of an evolutionary approach to evolve suitable neural networks for a given task Neural Networks In order to perform the prediction tasks described above, we use multi layer perceptron networks and a simple backpropagation learning rule. Then, we use an evolutionary strategy to change the following parameters of the network: • Number of hidden layers • Number of neurons in each layer • Transfer function employed by the neurons in each layer (neurons in the same layer will employ the same transfer function) In order to do this, we define a genetic code for the class of neural networks comprising an N digit binary bit string. In order to limit the optimisation search space, we arbitrarily limit the number of hidden layers to The number of neurons per layer is limited to The allowed transfer functions are linear, linear with bounds and hyperbolic tangens, as shown below: Thus, the number of neurons for each hidden layer can be represented by a four bit number, and the transfer function for the neurons in each layer by a two bit number, totalling a required six bits per layer. As there are up to 10 hidden layers, the total bitstring will be 60 bits long, and is represented as follows: Bit 0 Bit 59 Layer 0 Layer 1 Layer 2 Layer 3 Layer 4 Layer 5 Layer 6 Layer 7 Layer 8 Layer 9 Layer 4, Neuron bit 3 Layer 4, Neuron bit 2 Layer 4, Neuron bit 1 Layer 4, Neuron bit 0 Layer 4, Transfer bit 1 Layer 4, Transfer bit 0 Each layer can have up to 15 neurons, as given by the binary number [Neuron bit 3: Neuron bit 0]. If the bit string encodes zero neurons for that layer, it is interpreted as being non-existent. Also, as the bit string can encode for four transfer functions in each layer, as given by the binary number [Transfer bit 1: Transfer bit 0], but only three are employed, a bias is given towards the linear y = x transfer function, to be encoded in two of the possible four states. Thus, by changing the genes of a network, it will have a different topology, some more others less suitable for the tasks at hand. Optimisation In order to find an optimal network topology for the given tasks, we use an evolutionary strategy to evolve the genetic makeup of the network, which could also be regarded as a genetic algorithm without cross breeding. The Evolutionary Strategy The algorithm employed works as follows: 1. Produce a first generation of population size seven of random bit strings 2. Generate randomly initialised networks from the population of bit strings 3. Train the networks on a training set using backpropagation 4. Run the networks on the test data 5. A fitness function evaluates the fitness of each network, and the fittest network is kept for the next iteration, while the other ones are discarded. 6. The fittest network is cloned six times to refill the generation, and each of these clones is mutated randomly by inverting one of the 60 bits at random. 7. The process repeats at 3. until a maximum number of iterations has been performed, or a marginal or no improvement in the fitness can be achieved over several iterations A table that contains all the bitstrings that have already been evaluated is kept so as to avoid computing the same network topology multiple times. Here, we evaluate the fitness of the networks in two ways, depending on the task at hand. • For short term prediction, a part of the test signal series is used as input to the network, and the first value the network predicts is compared with the actual value of the series at that point. The cumulative error is found by summing the absolute difference between the predicted value and the actual value for all shifted versions of the actual signal as input to the network. The fitness of the network is the reciprocal of the cumulative error. This method evaulates the network’s ability to make short term predictions for a given pattern and number of inputs. (see below) Actual signal 1. 1 . . . . . . . N Neural network with inputs and output